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Abstract 



Cohen's popular book titled Statistical Power Analysis for the Behavioral Sciences, coupled with 
recent challenges to statistical significance, has made "effect size" one of the hottest methodological 
topics of our time. Cohen recommends specific levels of effect size for "small," "medium, and large 
effects. However, even Cohen acknowledged these values are relative to the specific content and method 
in a given research situation. The purpose of this study is to determine the probabilities of attaining 
varying magnitudes of standardized effect sizes by chance and when protected by a .05-level statistical 

test. 

Monte Carlo procedures were used to generate standardized effect sizes in a one-way ANOVA 
situation with 2 through 5, 6, 8, and 10 groups having selected sample sizes from 5 to 500. Within each 
of the 91 number of group and sample size configurations, 100,000 replications were generated from a 
distribution of normal deviates. For each data set, the effect size was computed along with a statistical 
test of hypothesis at the .05 level. For each n/k combination, the proportion of effect sizes exceeding 0. 1 
to 2.0 in increments of . 1 was computed for all cases and for those cases where the no difference 
hypothesis was rejected. 

There are trends that are common across all configurations. As the magnitude of effect size 
increases, the probability of getting such a difference by chance decreases as would be expected. Within 
a given number of samples situation, as sample size increases, as expected, the probability of getting such 
a difference by chance decreases. Within a given sample size, as the number of groups increases the 
probability of getting such a difference by chance increases. Another finding which is consistent across 
all configurations is that the significance test protected effect size probability is always equal to or less 
than the unprotected probability, in some cases dramatically so. It is clear that the addition of the 
significance test reduces the probability of finding a seemingly large effect size by chance. Such a 
protected effect size indicator could be an answer to the arguments posed by both those who protest 
against the use of the significance test and those who propose its use in judging the magnitude of an 
observed effect. 
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Avoiding Decision-Making by Chance: Protecting Effect Size Estimates 



Cohen's popular book titled Statistical Power Analysis for the Behavioral Sciences, coupled with 
recent challenges to statistical significance, has made "effect size" one of the hottest methodological 
topics of our time. The primary alternative that has been recommended to statistical hypothesis testing is 
the use of effect sizes (e.g.. Carver, 1978, 1993; Nix & Barnette, 1998). Cohen recommends specific 
levels of effect size for "small," "medium," and "large" effects. However, even Cohen acknowledged 
these values are relative to the specific content and method in a given research situation (Cohen, 1988). 
The purpose of this study is to determine the probabilities of attaining varying magnitudes of standardized 
effect sizes by chance and when protected by a .05-level statistical test. Results should provide empirical 
evidence of the wisdom of reporting an effect size estimate without first conducting a statistical test of the 

hypothesis. 

Background 

■ The concept of effect size has been around for many years. Cohen (1969) is generally credited 
with coining the term. However, the development of meta-analysis by Glass, Rosenthal and others in the 
1970s (e.g.. Glass, 1976; 1978; Glass & Hakstian, 1969; Rosenthal, 1976, 1978) and the popularity of a 
book on meta-analysis in 1981 (Glass, McGaw, & Smith) are the catalysts for the interest in the concept. 
Numerous publications followed on applications of effect size methodology (e.g.. Lynch, 1987; McLean, 
1983), methods for estimating effect size and its properties (e.g., Fowler, 1988; 1993; Gibbons, Hedeker, 
& Davis, 1993; Hedges, 1981, 1984; Huynh, 1989; Kraemer, 1983; Reichhardt & Gollob, 1987; Thomas, 
1986), extracting effect size estimates from existing studies (e.g.. Hedges, 1982; Snyder & Lawson, 
1993), and correcting effect size estimates (Snyder & Lawson, 1993). Another book by Wolf (1986), a 
general methodology for conducting meta-analysis, included the extraction and testing of effect sizes. 

Perhaps no one has had a greater impact on the use of effect sizes than Cohen (1988) through his 
books on power analysis. In these books, Cohen suggests general guidelines for levels of effect size. 
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These are .2 for small effect, .5 for medium effect, and .8 for large effect. However, even Cohen was 

concerned about proposing these as standards. He stated: 

The terms “small,” “medium,” and “large” are relative, not only to each other, but to the area of 
behavioral science or even more particularly to the specific content and research method being 
employed in any given investigation. In the face of this relativity, there is a certain risk inherent 
in offering conventional operational definitions for these terms for use in power analysis in as 
diverse a field of inquiry as behavioral science. This risk is nevertheless accepted in the belief 
that more is to be gained than lost by supplying a common conventional frame of reference which 
is recommended for use only when no better basis for estimating the ES index is available. 

(1988, p. 25) 

Cohen's concerns were cited by Wolf (1986) and suggest that effect sizes should be interpreted in 
context. Specifically, one possibility is to compare a given effect size to the median effect size of studies 
extracted from the professional literature in that specific context rather than use some arbitrary guideline. 
Wolf indicates that a .5 standard deviation improvement is often considered practically significant and 
that the general guidelines of the National Institute of Education's Joint Dissemination Review Panel 
require .33 effect size, but at times will accept .25 to establish educational significance. 

A broader debate on the use of statistical significance testing emerged from Cohen's power 
analysis books and other works. Kaufinan (1998) indicates that the "controversy about the use or misuse 
of statistical significance testing has been evident in the literature for the past 10 years and has become 
the major methodological issue of our generation" (p. 1). The debate has spawned at least two special 
issues of journals (Research in the Schools , McLean & Kaufinan, 1998; Journal of Experimental 
Education , Thompson, 1993) and dozens of other articles. The editorial policies of journals have been 
changed by the debate (e.g., AP A, 1994; Schafer, 1990, 1991; Thompson, 1987, 1997). 

The debate has ranged from those who recommend the elimination of statistical significance 
testing (e.g., Carver, 1978, 1993; Nix & Barnette, 1998) to those who staunchly support it (e.g., Frick, 
1996; Levin, 1993, 1998; McLean & Ernest, 1998). However, even those who defend statistical 
significance testing indicate that significant results should be accompanied by a measure of practical 
significance. The leading method of reporting practical significance is through the provision of an effect 
size estimate (Kirk, 1996; McLean & Ernest, 1998; Robinson & Levin, 1997; Thompson, 1996). 
Unfortunately, the criteria forjudging the practical significance of results based on effect size has 
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defaulted to the use of Cohen's (1988) guidelines that even Cohen has warned us about (1988, 1990). As 
Wolf (1986) noted, empirical standards forjudging effect size are needed. 

At least two recent studies have attempted to estimate effect sizes that might occur by chance. 

The first of these (Barnette & McLean, 1999, November), found that when no differences exist, effect 
sizes range from 0. 1 152 with 2 groups and an n of 100 to 1 .4044 with 10 groups and an n of 5, with a 
mean effect size of 0.4065. A second study by Barnette and McLean (2000, April) suggested that running 
statistical tests of hypothesis before computing effect sizes greatly reduced the number of spuriously high 
effect size values, with a mean effect size of 0.4065. 

Research Questions 

1 . What is the probability of attaining standardized effect sizes up to 2.0 by chance as functions of 
number and size of samples? 

2. What is the probability of attaining standardized effect sizes up to 2.0 that are also significant at .05 
by chance as functions of number and size of sample? 

3. How do these two probabilities compare? 

4. Is it feasible to use the Protected Effect Size Probability to evaluate the size of an observed 
standardized effect size? 

Methodology 

Monte Carlo methods were used to generate the data for this research using random normal 
deviates as the basis for sample means to be compared using one-way ANOVA. Standardized effect sizes 
were generated for 100,000 replications within each combination of number of groups of: 2, 3, 4, 5, 6, 8 
and 10 and sample sizes of: 5, 10, 15, 20, 25, 30, 40, 50, 60, 80, 100, 200, and 500, resulting in 9,100,000 
total replications. The standardized effect size was computed as the range of means divided by the root 
mean square error. In addition, the probability of the observed F statistic was evaluated using alpha 
values of .25, .10, .05, .01, and .001 so each observed effect size could be identified as being statistically 
significant in addition to observed standardized effect size. Data were generated using a program written 
in double-precision Quick-BASIC. Table 1 presents the observed probabilities of the F statistics 
generated by the Monte Carlo Program. Analysis of the raw data was conducted using several routines of 
SAS®. 
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For each number of samples and sample size configuration, there were two statistics computed: 
the observed F statistic probability and the standardized effect size. The analysis results were: the 
observed proportion of standardized effect sizes achieving or exceeding the effect size values of 0 to 2.0 
in steps of .1; and the observed proportion of standardized effect sizes achieving or exceeding the effect 
size values of 0 to 2.0 in steps of . 1 and being significant at an alpha level of .05. The latter statistic is 
what is referred to as the Protected Effect Size. Data were obtained which permit evaluation of other 
alpha levels, but this paper presents only those for alpha of .05. 

Results 

There is parsimony in presenting the answers to research questions 1, 2, and 3 for each number of 
samples configuration. There are trends that are common across all configurations. As the magnitude of 
effect size increases, the probability of getting such a difference by chance decreases as would be 
expected. Within a given number of samples situation, as sample size increases, as expected, the 
probability of getting such a difference by chance decreases. Within a given sample size, as the number 
of groups increases, the probability of getting such a difference by chance increases. Another finding 
which is consistent across all configurations is that the significance test protected effect size probability is 
always equal to or less than the unprotected probability, in some cases dramatically so. 

Tables relating to research question 1 (What is the probability of attaining standardized effect 
sizes up to 2.0 by chance as functions of number and size of samples?) are designated as Tables: 2, 4, 6, 8, 
10, 12, and 14, and their corresponding Figures are: 1, 3, 5, 7, 9, 11, and 13. Tables relating to research 
question 2 (What is the probability of attaining standardized effect sizes up to 2.0 that are also significant 
at .05 by chance as functions of number and size of sample?) are: 3, 5, 7, 9, 11, 13, and 15 and their 
corresponding figures are: 2, 4, 6, 8, 10, 12, and 14. Each Table and Figure represents a number of 
samples configuration with sample size included within the presentation. Each number of samples 
configuration will be presented separately and research question 3 (How do these two probabilities 
compare?) will be discussed. 
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The Two-Sample ( t Test) Situation 

Table 2 and Figure 1 present the proportions of the observed effect sizes which meet or exceed 
the effect size measures from 0 to 2, in units of . 1 . The probability of getting a small effect size (.2) 
ranges from .7636 when n- 5 to .002 1 when n= 500. It takes a sample size of about 200 to achieve less 
than a .05 probability. For a medium effect size (.5), the range of probabilities is from .453 1 when n= 5 
to .0000 when n= 500. It takes a sample size of greater than 30 to achieve less than .05 probability. For a 
large effect size (.8), the range of probabilities is .2419 when n= 5 to .0000 when n= 500. At a sample 
size of about 15 the probability drops below .05. Table 3 and Figure 2 present the proportions of the 
observed effect sizes that meet or exceed the effect size measures from 0 to 2, in units of . 1, while also are 
significant at .05 based on the significance test. A comparison of the unprotected probabilities and 
protected probabilities indicates that in the two-sample situation, they are the same. This will not be the 
case as the number of samples increases. 

The figures could be used to determine the probability of getting a given effect size by chance or 
getting one that also was significant at .05. For example, if an effect size was observed of .9 when 
samples of size 20 were used, the probability of getting this result would be about .0069, with the same 
probability had significance at .05 been required. Expansion of the tables and figures or, preferably, 
determination of equations to make this prediction, would be very useful in judging observed effect sizes. 

The Three-Sample Situation 

Table 4 and Figure 3 present the proportions of the observed effect sizes which meet or exceed 
the effect size measures from 0 to 2, in units of . 1, in the three-sample situation. The probability of 
getting a small effect size (.2) ranges from .9461 when n= 5 to .0052 when n= 500. It takes a sample size 
considerably higher than 200 to achieve less than a .05 probability. For a medium effect size (.5), the 
range of probabilities is from .7138 when n = 5 to .0000 when n = 500. It takes a sample size of greater 
than 40 to achieve less than .05 probability. For a large effect size (.8), the range of probabilities is .4376 
when n= 5 to .0000 when n= 500. At a sample size of between 15 and 20 the probability drops below .05. 
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Table 5 and Figure 4 present the proportions of the observed effect sizes that meet or exceed the effect 
size measures from 0 to 2, in units of .1, while also are significant at .05 based on the significance test. A 
comparison of the unprotected probabilities and protected probabilities indicates that in the three-sample 
situation, they are the same when samples are relatively large (50 or more): but at smaller sample sizes, 
the probability of the protected effect size is slightly lower than the unprotected effect size. 

The figures could be used to determine the probability of getting a given effect size by chance or 
getting one that also was significant at .05. For example, if an effect size of .75 was observed when 
samples of size 25 were used, the probability of getting this result would be about .015 had significance at 
.05 been required, but .03 if the significance test was not required. 

The Four-Sample Situation 

Table 6 and Figure 5 present the proportions of the observed effect sizes which meet or exceed 
the effect size measures from 0 to 2, in units of .1, in the four-sample situation. The probability of getting 
a small effect size (.2) ranges from .9890 when n= 5 to .0090 when n= 500. It takes a sample size 
considerably higher than 200 to achieve less than a .05 probability. For a medium effect size (.5), the 
range of probabilities is from .8582 when n= 5 to .0000 when n= 500. It takes a sample size of greater 
than 50 to achieve less than .05 probability. For a large effect size (.8), the range of probabilities is .5985 
when n= 5 to .0000 when n= 500. At a sample size of between 20 and 25 the probability drops below .05. 
Table 7 and Figure 6 present the proportions of the observed effect sizes that meet or exceed the effect 
size measures from 0 to 2, in units of . 1 , while also are significant at .05 based on the significance test. A 
comparison of the unprotected probabilities and protected probabilities indicates that in the four-sample 
situation, they are the same when samples are very large (100 or more) but at smaller sample sizes, the 
probability of the protected effect size was increasingly lower than the unprotected effect size as the effect 
size standard increased. For example, in the n= 5 situation an effect size of 1.7 had a probability that was 
not lower than .05 (.0702) but at the same difference magnitude the protected probability was .0480. 

Using the figures to determine the probability of getting a given effect size by chance or getting 
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one that also was significant at .05, if an effect size was observed of .65 when samples of size 30 were 
used, the probability of getting this result would be about .042 had significance at .05 been required, but it 
would not have been higher than .05 if the significance test was not required. 

The Five, Six, and Eight Sample Situations 

Review of Tables 8, 9, 10, 1 1, 12, and 13 and Figures 7, 8, 9, 10, 1 1, and 12 present results very 
similar to those found in the three and four-sample situations. The probabilities are slightly higher in all 
cells, but the comparative patterns are the same. 

The Ten-Sample Situation 

Table 14 and Figure 13 present the proportions of the observed effect sizes which meet or exceed 
the effect size measures from 0 to 2, in units of . 1, in the ten-sample situation. The probability of getting 
a small effect size (.2) ranges from 1 .0000 when n= 5 to .053 1 when n= 500. It takes a sample size higher 
than 500 to achieve less than a .05 probability. For a medium effect size (.5), the range of probabilities is 
from .9984 when n= 5 to .0000 when n= 500. It takes a sample size of greater than 80 to achieve less 
than .05 probability. For a large effect size (.8), the range of probabilities is .9552 when n= 5 to .0000 
when n= 500. At a sample size greater than 30 the probability drops below .05. Table 15 and Figure 14 
present the proportions of the observed effect sizes that meet or exceed the effect size measures from 0 to 
2, in units of . 1, while also are significant at .05 based on the significance test. A comparison of the 
unprotected probabilities and protected probabilities indicates that in the ten-sample situation, they are 
much lower across all sample size configurations. For example, in the n= 40 situation an effect size of 
0.6 had a probability that was not lower than .05 (. 1 835) but at the same difference magnitude the 
protected probability was .0478. 



Conclusions 

This paper clearly demonstrates that using effect size in research without a statistical test of 
hypothesis is a very risky undertaking. It is quite true that statistical tests of hypothesis provide only one 
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type of information used for decision-making. However, interpreting effect size results by themselves can 
lead to just as poor, if not poorer, decisions than using only statistical tests of hypothesis. In general, we 
concur with McLean and Ernest (1998), who conclude that "statistical significance testing must be 
accompanied by judgments of the event's practical significance and replicability" (p. 21). However, we 
would add that a measure of practical significance such as effect size should be accompanied by a 
statistical test of hypothesis. 

Is it feasible to use the Protected Effect Size Probability to evaluate the size of an observed 
standardized effect size? It is clear that the addition of the significance test reduces the probability of 
finding a seemingly large effect size by chance. Such a protected effect size indicator could be an answer 
to the arguments posed by both those who protest against the use of the significance test and those who 
propose its use in judging the magnitude of an observed effect. 

Further Monte Carlo work which expands the data set and work on determining equations which 
could be used rather than the use of cumbersome tables could go a long way to make meaningful 
difference decision-making more useful and scientifically sound. Further research needs to include 
sampling from populations not meeting the normality and homogeneity assumptions, as well as the use of 
unequal sample sizes. 
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Table 1. Observed F Statistics of Total Monte Carlo Runs 



Expected Probability 


Observed Probability 


.25 


.24907 


.10 


.09986 


.05 


.05011 


.01 


.01009 


.001 


.00098 
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Table 2. Proportion of Standardized Effect Sizes Exceeding Selected Values with no Significance Test, K= 
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Table 4. Proportion of Standardized Effect Sizes Exceeding Selected Values with no Significance Test, K= 
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Table 14. Proportion of Standardized Effect Sizes Exceeding Selected Values with no Significance Test, K= 10 
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Figure 1. Probability of Effect Size by Chance, 
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Figure 2. Protected Probability, Alpha= .05, K= 
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Figure 3. Probability of Effect Size by Chance, 



CO 

II 



* 



o in o in o 

LO t— t— CM CM CO 

It II II II II II 

C C C C C C 



O O O 
O O O O O O O 

rf UO to CO T- CM LD 



JL JL II II II It II II 

ccccccccccccc 




LO 

co 



oh 

CM 



CO 




Figure 4. Protected Probability, Alpha= .05, K= 
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Figure 5. Probability of Effect Size by Chance, 
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Figure 6. Protected Probability, Alpha= .05, K= 
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Figure 7. Probability of Effect Size by Chance, 
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Figure 8. Protected Probability, Alpha= .05, K= 
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Figure 9. Probability of Effect Size by Chance, 
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Figure 10. Protected Probability, A!pha= .05, K= 
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Figure 1 1 . Probability of Effect Size by Chance, 
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Figure 12. Protected Probability, Alpha= .05, 
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Figure 13. Probability of Effect Size by Chance, 
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Figure 14. Protected Probability, Alpha= 
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